Adadb: Adaptive Diff-Batch Optimization Technique for Gradient Descent

Gradient descent is the workhorse of deep neural networks. has disadvantage slow convergence. The famous way to overcome convergence use momentum. Momentum effectively increases learning factor gradient descent. Recently, many approaches have been proposed control momentum for better optimization towards global minima, such as Adam, diffGrad, and AdaBelief. Adam decreases by dividing it with square root moving averages squared past gradients or second moment. sudden decrease in moment often results overshoot from minima then settle at closest minima. DiffGrad this problem using a friction constant based on difference current immediate Adam. further AdaBelief adapts step size according belief direction. Another fast increase batch adaptively. This paper proposes new technique named adaptive diff-batch adadb that removes overshooting combines methods rate. uses three differences rather than one diffGrad condition decide constant. outperformed optimizers synthetic complex non-convex functions real-world datasets.